Strong Scaling Study with PackageCompiler

Adam Lyon, Muon g-2 IRMA Analysis, Fermilab @ Home, November 2020

This notebook examines strong scaling properties of Julia IRMA jobs making just one plot. See the strongScaling.jl notebook for background. For this notebook, the PackageCompiler.jl was run to compile functions needed by the job. Doing so improved the Julia startup time significantly.

This notebook answers issue Analyze results from Strong Scaling jobs #18 and code may be found in PR #20. This file is IRMA/analyses/018_StrongScaling/StrongScaling_pkgc.jl.

What is this notebook?

This is a Pluto.jl notebook and the code here is written in Julia. This is like a Jupyter notebook, but with important differences. The most important difference is that the results appear above the code. Another important difference is that Pluto.jl notebooks are reactive. This means that unlike Jupyter notebooks, Pluto.jl notebooks are always in a consistent state. The notebook keeps track of the cell-to-cell dependencies and when a cell changes, the dependent cells update at the same time. This means that while you are looking at a static html representation of the notebook, you can be assured that the notebook is consistent and up-to-date. You'll see that some results have a little triangle next to them. Clicking on that will open an expanded view of the results.

The organization of this notebook is that the main results are replicated at the top, with discussion, in the Results section. The plots are stored in variables which you can see below the plot. You can look in the Code section, which has all code for this notebook, to see how the plot was made.

Introduction

This notebook examines strong scaling properties of my Julia IRMA jobs that make one plot of energy of clusters. Nearly all of the runs used the maximum 32 physical cores on each node (tasks). I ran with 2, 3, 4, 5, 6, 8, 10, 12, 15, and 20 nodes. To look for conflicts between tasks on nodes, I ran 5 nodes with 2, 4, 8, 16, and 32 tasks.

On 11/2, I ran IRMA/jobs/003_StrongScaling/strongScalingJob.jl that was compiled by PackageCompiler.jl from commit 823ce57 of master (PR #22). See this comment in issue #3.

I ran over era 2D data, which amounted to approximately 16B rows. The total HDF5 file size is 250 GB.

PackageCompiler.jl (see documentation) allows for precompilation of functions that are actually used by the Julia script. A shared object (.so) file is loaded at Julia start time. Using PackageCompiler.jl significantly improves the time it takes to load packages and functions do not need to be JIT compiled, speeding up the code. In MPI environment, using PackageCompiler.jl is ideal to avoid each rank doing its own JIT compilation. The downside of PackageCompiler.jl is that the compiled code in the shared object file is fixed, and so if you update packages to new versions, you may have a mistmatch. But given that the environment is pretty carefully controlled, this isn't much of a problem.

This notebook is an update to a previous analysis Pluto notebook at IRMA/analyses/018_StrongScaling/StrongScaling.jl. That analysis did not use PackageCompiler.jl and there seemed to be significant time loading packages and compiling. This notebook using PackageCompiler.jl shows signficant stability and improvement in timings.

There are occaisonal anomalies. I did a 10 node job that took an extremely long time, running it again produced more normal looking timings.

Results

There are several types of results.

Histogram comparison

This was done in the previous notebook without PackageCompiler.jl. I did not re-run this analysis.

MPI Timing information

I record the time in the job with an IRMA Stopwatch. The stopwatch uses MPI.Wtime under the hood. The times are recorded as follows.

Label	Meaning
start	After packages are loaded, functions are defined, and MPI.Init call
openedFile	After the h5open statement
openedDataSet	After the energy dataset is opened (but no data read yet)
determineRanges	After the ranges to examine are determined with partitionDS
readDataSet	After the dataset is read (this reads the actual data for the rank)
makeHistogram	After the data is histogrammed
gatheredHistograms	After all the histograms have been gathered to the root rank
reducedHistograms	After the histograms have been reduced to the root rank
gatherRankLogs	After the rank's log info has been gathered to the root rank

Timing of anything before start is not recorded.

Let's look at the timing plots. Note that you can see all the timing plots in the Code section. Some representative plots will be reproduced here.

Here is a run using three nodes (and 32 ranks per node).

Each rank reads about 175,375,305 rows (or one more). These timings are slightly better than without PackageCompiler.jl (package loading is not shown here).

And here's ten nodes...

Again, we see that some nodes are faster than others for reading.

Here's a box and whisker plot of total MPI time...

Let's just look at the reading time and see if that scales appropriately.

There's a bit of variation. Look at the plots in the Code section to see why.

Let's now concentrate on seeing if the reading time scales. We'll take the mean read time for each run...

The expected time is anchored at the 2 node run. So a four node run should be twice as fast if the scaling were perfect. And what we see is pretty close.

10 rows × 4 columns

	numNodes	readDataSet_maximum	readDataSet_mean	expectedFromMean
	Int64	Float64	Float64	Float64
1	2	36.7791	34.7144	34.7144
2	3	25.3889	23.6609	23.1429
3	4	18.845	17.7989	17.3572
4	5	15.5825	14.2538	13.8858
5	6	12.8665	11.9569	11.5715
6	8	9.9717	9.24282	8.6786
7	10	9.3017	7.8274	6.94288
8	12	8.17051	6.70655	5.78573
9	15	6.33904	5.63782	4.62859
10	20	6.51274	5.26359	3.47144

We can also look at reducing the number of ranks per node to see if there's any significant i/o contention. Here is the read time for 5 nodes but varying the number of ranks per node.

Let's calculate the fraction of time in contention; that is the difference between the measured read time and the expected time divided by the measured read time.

So the above plot suggests that fewer ranks per node is more efficient. When we ask for 32 nodes, we are possibly wasting about 30% of the read time in i/o contention. Here's a table,...

5 rows × 6 columns

	numTasks	readDataSet_maximum	readDataSet_mean	expectedFromMean	diff	diffPerc
	Int64	Float64	Float64	Float64	Float64	Float64
1	2	161.71	161.093	161.093	0.0	0.0
2	4	83.0948	82.5291	80.5466	1.98251	2.4022
3	8	46.1693	45.4499	40.2733	5.17663	11.3897
4	16	27.7527	25.9477	20.1367	5.81102	22.3951
5	32	15.5825	14.2538	10.0683	4.18546	29.3638

Remember, there are always 5 nodes in the job.

Accounting Information

We can extract accounting information from the SLURM batch system to see how long the jobs took.

The total batch time is the total wall clock time for the job. The total Julia time is how much time the srun julia ... command took within the batch script. The MPI Julia time is the amount of time that was recorded within MPI. Note that the latter does not include the time it took to start Julia and load packages as well as the time to write the results to disk.

The difference between the total batch and julia times are likely due to setting up the requested number of nodes. This time appears to be significant when requesting a large number of nodes. The total Julia time shape is puzzling, though it appears it is most efficient around six or seven nodes. Not clear why it increases and why it appears to turn over before twenty nodes.

Conclusions

These results are improved and more stable by using PackageCompiler.jl. Things that I conclude from this study.

• Use PackageCompiler.jl

• 32 tasks per node seems to involve significant i/o contention

• Configuring a large number of nodes takes a signifcant amount of time.